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Abstract: Modeling the behavior of the cognitive 
architecture in the context of social simulation using 
statistical methodologies is currently a growing 
research area. Normally, a cognitive architecture for 
an intelligent agent involves artificial computational 
process which exemplifies theories of cognition in 
computer algorithms under the consideration of state 
space. More specifically, for such cognitive system 
with large state space the problem like large tables 
and data sparsity are faced. Hence in this paper, we 
have proposed a method using a value iterative 
approach based on Q-learning algorithm, with 
function approximation technique to handle the 
cognitive systems with large state space. From the 
experimental results in the application domain of 
academic science it has been verified that the 
proposed approach has better performance compared 
to its existing approaches. 

Keywords: Cognitive architecture, Social Simulation, 
Reinforcement learning, Function approximation. 

I. INTRODUCTION 

A cognitive architecture specifies the underlying 
infrastructure for an intelligent system. Briefly, 
architecture includes those aspects of a cognitive agent 
that are constant over time and across different 
application domains. These typically include: 

1. The short-term and long-term memories that store 
content about the agent's beliefs, goals, and 
knowledge; 

2. The representation of elements that are contained in 
these memories and their organization into larger- 
scale mental structures; 

3. The functional processes that operate on these 
structures, including the performance mechanisms 
that utilize them and the learning mechanisms that 
alter them. 

Because the contents of an agent's memories can 
change over time, one would not consider the 
knowledge and beliefs encoded therein to be part of 
that agent's architecture. As different programs can run 



on the same computer architecture, so the different 
knowledge bases and beliefs can be interpreted by the 
same cognitive architecture. There is also a direct 
analogy with a building's architecture, which consists 
of permanent features like its foundation, roof, and 
rooms, rather than its furniture and appliances, which 
one can move or replace. 

II. EXAMPLES OF COGNITIVE ARCHITECTURES 

A. ACT-R 

ACT-R [7] [8] is the latest in a family of cognitive 
architectures, concerned primarily with modeling 
human behavior, that has seen continuous development 
since the late 1970s. ACT-R is organized into a set of 
modules, each of which processes a different type of 
information. These include sensory modules for visual 
processing, motor modules for action, an intentional 
module for goals, and a declarative module for long- 
term declarative knowledge. Each module has an 
associated buffer that holds a relational declarative 
structure (often called 'chunks', but different from 
those in Soar). Taken together, these buffers comprise 
ACT-R' s short-term memory. 

B. SOAR 

Soar [4][5][1] is a cognitive architecture that has 
been under continuous development since the early 
1980s. Procedural long-term knowledge in Soar takes 
the form of production rules, which are in turn 
organized in terms of operators associated with 
problem spaces. Some operators describe simple, 
primitive actions that modify the agent's internal state 
or generate primitive external actions, whereas others 
describe more abstract activities. For many years, Soar 
represented all long-term knowledge in this form, but 
recently separate episodic and semantic memories 
have been added. The episodic memory [2] holds a 
history of previous states, while semantic memory 
contains previously known facts. 

C. ICARUS 

ICARUS is a more recent architecture [14] that 
stores two distinct forms of knowledge. Concepts 
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describe classes of environmental situations in terms of 
other concepts and percepts, whereas skills specify how 
to achieve goals by decomposing them into ordered sub 
goals. Both concept and skills involve relations among 
objects, and both impose a hierarchical organization on 
long-term memory, with the former grounded in 
perceptions and the latter in executable actions. 
Moreover, skills refer to concepts in their heads, their 
initiation conditions, and their continuation conditions. 

D. PRODIGY 

PRODIGY [6] is another cognitive architecture that 
saw extensive development from the middle 1980s to 
the late 1990s. This framework incorporates two main 
kinds of knowledge. Domain rules encode the 
conditions under which actions have certain effects, 
where the latter are described as the addition or deletion 
of first-order expressions. These refer both to physical 
actions that affect the environment and to inference 
rules, which are purely cognitive. In contrast, control 
rules specify the conditions under which the 
architecture should select, reject, or prefer a given 
operator, set of operator bindings, problem state, or 
goal during the search process. 

PRODIGY'S explanation-based learning module 
constructs control rules based on its problem-solving 
experience [20]. Successful achievement of a goal after 
search leads to creation of selection or preference rules 
related to that goal and to the operators whose 
application achieved it. In addition, PRODIGY 
includes separate modules for controlling search by 
analogy with earlier solutions [11] learning operator 
descriptions from observed solutions or 
experimentation [21], and improving the quality of 
solutions [12]. Although most research in this 
framework has dealt exclusively with planning and 
problem solving, PRODIGY also formed the basis for 
an impressive system that interleaved planning and 
execution for a mobile robot that accepted 
asynchronous requests from users [10]. 

E. THE CLARION 

CLARION is an integrative cognitive architecture 
consisting of several distinct sub systems [16] [18] [15]. 
These include the action-centred subsystem (ACS), the 
non-action-centred subsystem (NACS), the 
motivational subsystem (MS), and the metacognitive 
subsystem (MCS). The ACS controls actions, whether 
for external physical movements or internal mental 
operation. The NACS maintains general knowledge, 
either implicit or explicit. The MS provides under-lying 
motivations for perception, action, and cognition in 
terms of impetus and feedback (for example, indicating 
whether outcomes are satisfactory or not). The MCS 
monitors, directs, and modifies the operations of the 
ACS dynamically, as well as the operations of all the 
other subsystems. 



Modeling the behavior of the cognitive architecture 
in the context of social simulation using statistical 
methodologies is currently a growing research area. 
Normally, a cognitive architecture for an intelligent 
agent involves artificial computational process which 
exemplifies theories of cognition in computer 
algorithms under the consideration of state space. 
More specifically, for such cognitive system with large 
state space the problem like large tables and data 
sparsity are faced. Hence in this paper we have 
proposed a method using a value iterative approach 
based on Q-learning method with function 
approximation to handle the cognitive systems with 
huge state space. From the experimental results in the 
domain of academic science it has been proceeded that 
to proposed approach results better performance 
compared to its existing approach. 

III. CAPABILITIES OF COGNITIVE 
ARCHITECTURES 

Any intelligent system is designed to engage in 
certain activities that, taken, together, constitute its 
functional capabilities. This section, discusses the 
varied capabilities that a cognitive architecture can 
support. Although only a few abilities, such as 
recognition and decision making, are strictly required 
to support a well-defined architecture, the entire set 
seems required to cover the full range of human-level 
intelligent activities. 

• Recognition and Categorization 

• Decision Making and Choice 

• Perception and Situation Assessment 

• Prediction and Monitoring 

• Reasoning and Belief Maintenance 

• Execution and Action 

• Interaction and Communication 

• Remembering, Reflection and Learning 

• Problem solving and Planning 

IV. COGNITIVE MODELS 

Cognitive models attempt to represent the users as 
they interact with a system, modeling aspects of their 
understanding, knowledge, intentions or processing. 
Cognitive models are divided into three categories. 

• The first described the hierarchical structuring of 
the user's task and goal structure. Eg. GOMS 
(Goals, Operators, Method, Selection) CCT 
(Cognitive Complexity Theory). 

• The second model concerned with Linguistic and 
grammatical models, which emphasized the user's 
understanding of the user system - dialog. Eg. BNF 
(Backus Naur Form), TAG (Task Action 
Grammar). 
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V. 



Third model based on the more solid understanding 
of the Human Motor System. Eg. KLM (Keystroke 
- Level Model). 

EXISTING A COGNITIVE ARCHITECTURE IN 
SOCIAL SIMULATION 



One application of CLARION to social simulation 
is in understanding organizational decision making and 
the interaction between organizational structures and 
cognitive factors in affecting organizational decision 
making [Sun, 04]. 

In terms of organizational structures, there are two 
major types: (1) teams, in which agents act 
autonomously, individual decisions are treated as 
votes, and the organizational decision is the majority 
decision; and (2) hierarchies, which are characterized 
by agents organized in a chain of command, such that 
information is passed from subordinates to superiors, 
and the decision of a superior is based solely on the 
recommendations of his/her subordinates. In addition, 
organizations are distinguished by the structure of 
information accessible by each agent. Two varieties of 
information access are: (1) distributed access, in which 
each agent see a different subset of attributes (no two 
agents see the same sub-set of attributes), and (2) 
blocked access, in which several agents see exactly the 
same subset of attributes. 

Several simulation models were considered in [9]. 
The experiments by [9] were done in a 2 x 2 fashion 
(organization x information access). In addition, 
human data for the experiment were compared to the 
results of the four models [9]. 

In their work, the agent models used were very 
simple, and the "intelligence" level in these models 
was low. Moreover, learning in these simulations was 
rudimentary: there was no complex learning process as 
one might observe in humans. With these 
shortcomings in mind, it is worthwhile to undertake a 
simulation that involves more complex agent models 
that more accurately capture human performance. 
Moreover, with the use of more cognitively realistic 
agent models, one may investigate individually the 
importance of different cognitive capacities and 
process details in affecting organizational performance 
[19]. 

Hence, a simulation with CLARION used for 
modelling individual agents in an organization was 
conducted [19]. The results closely accord with the 
patterns of the human data, with teams outperforming 
hierarchical structures, and distributed access proving 
superior to blocked access. 

Also, as in humans, performance is not grossly 
skewed towards one condition or the other, but is 
roughly comparable across all conditions, unlike some 
of the simulation results from [9]. The match with the 



human data is far better than in the simulations 
conducted in [9]. The better match is due, at least in 
part, to a higher degree of cognitive realism in this 
simulation. See [19] for further details, including the 
interesting effects of varying cognitive parameters. 

Another application of CLARION to social 
simulation is in capturing and explaining the essential 
process of publication in academic science and its 
relation to cognitive processes [3]. Science develops in 
certain ways. In particular, it has been observed that 
the number of authors contributing a certain number of 
articles to a scientific journal follows a highly skewed 
distribution, corresponding to an inverse power law. In 
the case of scientific publication, the tendency of 
authorship to follow such a distribution was known as 
Lotka's law. 

Herbert Simon developed a simple stochastic 
process for approximating Lotka's law. One of the 
assumptions underlying this process is that the 
probability that a paper will be published by an author 
who has published i articles is equal to a/i , where a is 
the constant of proportionality. Using Simon's work as 
a starting point, [13] attempted to model Lotka's law. 
He obtains his simulation data based on some very 
simplified assumptions and a set of mathematical 
equations. To a significant extent, Gilbert's model is 
not cognitively realistic. The model assumes that 
authors are non-cognitive and interchangeable; it 
therefore neglects a host of cognitive phenomena that 
characterize scientific inquiry (e.g., learning, 
creativity, evolution of field expertise, etc.). 

Using a more cognitively realistic model, one can 
address some of these omissions, as well as exploring 
other emergent properties of a cognitively based model 
and their correspondence to real-world phenomena. 
The results of the simulation based on CLARION [3] 
are shown along with results (reported by [13]) for 
Chemical Abstracts and Econometrica, and estimates 
obtained from previous simulations by [3]. The tables 
indicate number of authors contributing to each 
journal, by number of papers each has published. More 
specifically, for such cognitive system with large state 
space the problem like large tables and data sparsity 
are faced. Hence in this paper we have proposed a 
method using a value iterative approach based on Q- 
learning algorithm with function approximation 
technique to handle the cognitive systems with large 
state space. 

VI. PROPOSED WORK 

Function approximation and feature based method 
A. Problem Specification 

We consider the following form of RL task for the 
learning agent. 
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What is given? 

• Set S of possible states. 

• Set A of possible actions. 

• Unknown transition function 8: S XA -> S 

• Reward function R, which is 1 in goal state and o 
else where goal states are terminal. 

What to find? 



An optimal policy Yl* = S ^> A for selecting the 
next action ai based on current observed states. 

It may be generally very difficult to learn a Q- 
Function perfectly. We often expect learning 
algorithms to set only some approximation to the 
target function. 

B. Proposed Architecture 



ACS 



NACS 



action-centered 

explicit representation 



action-centered 
implicit representation 



non-action-centered 
explicit representation 



non-action-centered 
implicit representation 




Figure 1: Clarion Architecture with Function Approximation 



We will use function approximation here and learn 
a representation of the Q-Function as a linear function 
of combinations of features, where the features 
describe a state. In other words, we will translate a 
state S into the set of features//, f2,....fn. Where n is 
the number of features. 

We have set of Q- functions: 

Q a (s,a)= 9 1 af 1 + - + G n af n 

The update rule is 

ff* k = ff* k + a [r + y max a , Q° l ( s' , a') - Q a (s,a)] dQ a (s,a) 

dff* k 

(1) 

Algorithm The RL Algorithm with Function 

Algorithm 

initialize all thetas to 

repeat 

Generate a starting state s 

i = 
repeat 
Choose an action a i? using the policy obtained from 



The current values of thetas 
Execute action a t , observe R and s i+1 
i=i+l 

until St is terminal(i.e., a goal state) 
for j = i - 1 to do 

Update the value of ff* k for all taken actions using (1) 
end for 
until no more episodes 

VII. EVALUATION AND RESULTS 

Science develops itself in particular ways. In particular 
the number of authors contributing a certain number of 
articles to scientific journals tends to follow a highly 
skewed distribution, corresponding to an inverse 
power law. In the case of scientific publications, the 
tendency of authorship to follow such a distribution is 
known as Lotka's law. Herbert Simon developed a 
simple stochastic process for approximating Lotka's 
Law. One of the assumptions underlying this process is 
that the probability that a paper will be published by an 
author who has publiched i articles is ali k , where a is a 
constant of proportionality. Using Simon's work as a 
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starting point, Gilbert attempted to model Lotka's 
Law. He obtained his simulation data on the basis of 
some simplified assumptions and a set of mathematical 
equations. To a significant extent, Gilbert's model was 
not cognitively realistic. The model assumed that 
authors were non-cognitive and interchangeable; it 
therefore neglected a host of cognitive phenomena that 
characterized scientific inquiry. 

In clarion simulation they have introduced a 
cognitive approach by comparing the previous results. 
In our model the number of articles per author 
reflected the cognitive ability of an author, as opposed 
to being based on auxiliary assumptions such as those 
made by Gilbert. 



Our model is using learning and function 
approximation is able to give a good match to the 
results of clarion simulation. Using function 
approximation we have got better output than the 
previous results. That is, we put more distance 
between mechanisms and outcomes which made it 
more difficult to obtain a match with the human data. 
Even though, being able to match the human data 
reasonably well shows the power of our approach. 

The proposed method data for the two journals 
could be fit to the power curve f(i) = a/i, resulting in 
an excellent match. Table 1 and Table 2 clearly show 
the good results achieved from the proposed method. 



Table 1: Number of Authors Contributing to Chemical Abstracts. 



No. of Papers 


Actual 


Simon's Estimate 


Gilbert's simulation 


CLARION simulation 


Proposed Method 


1 


3991 


4050 


4066 


3803 


3850 


2 


1059 


1160 


1175 


1228 


1238 


3 


493 


522 


526 


637 


646 


4 


287 


288 


302 


436 


444 


5 


184 


179 


176 


245 


252 


6 


131 


120 


122 


200 


206 


7 


113 


86 


93 


154 


160 


8 


85 


64 


63 


163 


168 


9 


64 


49 


50 


55 


60 


10 


65 


38 


45 


18 


20 


1 1 or more 


419 


335 


273 


145 


150 


Table 2: Number of authors contributing to Econometrica 


No. of Papers 


Actual 


Simon's Estimate 


Gilbert's simulation 


CLARION simulation 


Proposed approach 


1 


436 


453 


458 


418 


430 


2 


107 


119 


120 


135 


138 


3 


61 


51 


51 


70 


72 


4 


40 


27 


27 


48 


50 


5 


14 


16 


17 


27 


28 


6 


23 


11 


9 


22 


23 


7 


6 


7 


7 


17 


19 


8 


11 


5 


6 


18 


19 


9 


1 


4 


4 


6 


9 


10 





3 


2 


2 


4 


1 1 or more 


22 


25 


18 


16 


20 
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Figure 2: Proposed method result for different setting 
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Figure 3: Proposed method with CLARION 
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Figure 4: Proposed method result for different setting 
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Figure 5: Proposed method with clarion. 

Note that, in this simulation, the number of papers 
per author reflected the cognitive ability of an author, 
as opposed to methods based on auxiliary assumptions 
such as those made by [13]. This explains, in part, the 
greater divergence of these results from the human 
data: whereas Gilbert's simulation consists of equations 
selected to match the human data. This approach relies 



on much more detailed and lower-level mechanisms 
namely, a cognitive agent model that is generic rather 
than being task-specific. The result of the CLARION 
based simulation is therefore emergent, and not a result 
of specific and direct attempts to match the human data. 
That is, one put more distance between mechanisms 
and outcomes, which makes it harder to obtain a match 
with the human data. Thus, the fact that it was able to 
match the human data reasonably well shows the power 
of this cognitive architecture based approach. 

VIII. CHALLENGES FACING COGNITIVE 
SOCIAL SIMULATION 

An important development in the social science has 
been that of agent-based social simulation (ABSS). 
This approach consists of instantiating a population of 
agents, allowing the agents to run, and observing the 
interactions between them. The use of agent-based 
social simulation as a means for computational study of 
societies mirrors the development of cognitive 
architectures in cognitive science. 

The two fields of social simulation and cognitive 
architectures can be profitably integrated. This is an 
important challenge. As it has been argued before, 
social processes ultimately rest on the choices and 
decisions of individuals, and thus understanding the 
mechanisms of individual cognition can lead to better 
theories describing the behavior of aggregates of 
individuals. 

At the same time, by integrating social simulation 
and cognitive modeling, one can arrive at a better 
understanding of individual cognition. Traditional 
approaches to cognitive modeling have largely ignored 
the potentially decisive effects of socially acquired and 
disseminated knowledge (including language, norms, 
and so on). By modeling cognitive agents in a social 
context, one can learn more about the socio cultural 
processes that influence individual cognition. 

The most fundamental challenge in this regard is to 
develop better ways of conducting detailed social 
simulation based on cognitive architectures as basics. 
Although some cognitive details may ultimately prove 
to be irrelevant, this cannot be determined a priori, and 
thus simulations are useful in determining which 
aspects of cognition can be safely abstracted away 
building blocks. This is not an easy task. Although 
some initial work has been done (e.g., [17][19] much 
more work is needed. 

There is also the challenge of computational 
complexity. Social simulation could involve a large 
number of agents, up to thousands. Computational 
complexity is thus already high, even without involving 
cognitive architectures as agent models. To incorporate 
cognitive architectures into social simulation, one has 
to deal with a great deal of added complexity. 
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IX. CONCLUSION 

We have proposed here a method using a 
value iterative approach based on Q-learning algorithm 
with function approximation to handle the cognitive 
systems with large state space. From the experimental 
results in the domain of academic science it has been 
proved that the results of proposed approach are better 
as compared to its existing approach. These results and 
algorithms can be very useful to the researchers those 
who are in academic science and cognitive 
architecture. 

X. FUTURE WORK 

Emotions play a central role in human 
behavior, yet few systems offer any account of their 
purpose or mechanisms. There is a need of new 
architectures that exhibit emotion in ways that link 
directly to other cognitive processes and that modulate 
an intelligent behavior. Most architecture incorporate 
some form of learning, but none have shown the 
richness of improvement that humans demonstrate. 
There is a need of more robust and flexible learning 
mechanisms that are designed for extended operation 
in complex, unfamiliar domains. 
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